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Introduction 

Economic educators have long emphasized that the teaching and learning of economics should 
result in the increased willingness and ability of students to use the discipline of economics in solving 
problems, both personal and social. Given this emphasis, the focus of much research in economic education 
has been on the factors affecting student uncsrstanding and application of economic concepts. However, 
recent research calls into question the adequacy of this approach as a means to explore the relationship 
between educating for economic understanding and the goal of better economic problem solving. Some 
researchers are investigating the literature concerning schemata and expert problem solving drawn from 
cognitive psychology (Miller and VanFosscn, 1994; VanSickle, 1992 and VanSickle and Hoge, 1991), which 
seems to hold promise for linking economic education and economic problem solving. 

A previous study by Miller and VanFossen (1994) has described key differences between expert and 
novice problem solvers in economics. However, explicating the distinctions between novices and experts 
only begins the quest to understand how one acquires exjjertise in economic problem solving and, further, to 
promote expertise through economic education. A crucial step is to understand relative expertise, that is 
differences in expertise not just between experts and novices, but among individuals at various points along 
the path to expertise. Indeed, as the characteristics of relative expertise in economic problem solving 
become known, it may be possible to design curriculum in economic education that might better assist 
students in rrwving closer to this level of problem solving ability. 

This article reports preliminary research into the nature of relative expxjrtise in economic problem 
solving. The first section briefly describes why such research is needed in the context of research on expert 
and novice problem solving. It also presents tlie problem explored in this study in the context of the existing 
research. Subsequent sections present the methods, results, and conclusions of the study. 

Context of the Research and the Problem 
Economic Education and Problem Solving 

Since at least the 1961 Report of the National Task Force on Economic Educatioti, economic educators 
have long contended that economic education should develop the ability of students to use economics to 
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solve problems, a goal that has been reiterated many times over the years (Saunders, et al., 1993; Buckles, 
1989; Miller, 1988; Schug, 1985; and Symmes and Gilliard, 1981). Moreover, at least one important result of 
this enhanced economic problem solving ability should be more effective and responsible citizenship (Miller, 
1991 and 1989; and Brenneke, 1992). Indeed, increasing democratic citizens abilities to engage in more 
effective economic problem solving is the raison d 'etre of economic education. 

Thus the rationale for economic education posits a chain: that more and better economic education 
will lead to greater student understanding of economics which will lead to better economic problem solving 
resulting in more r isponsible citizenship (Miller and VanFossen, 1994). Of the links in the chain, the first ~ 
the relationship of economic education to greater student understanding of economics - has received the 
most attention, with particular emphasis on studies of the impact of economic instruction on scores of 
students on standardized economics tests (See Walstad and Soper, 1991, for numerous examples). Until very 
recently, research on the relationship of understanding to problem solving, the second link, was largely 
confined to studies comparing students' aned economists' opinions on various economic issues. The third 
link has been explored not at all. 

There is, however, reason to question whether competence in economics should be equated with 
understanding of economic concepts and, further, whether knowledge of economic concepts alone is 
sufficient, in and of itself, to lead to greater expertise in economic problem solving. Research in expert and 
novice problem solving and schema theory suggest the need for a more complete conceptualization of 
economic knowledge. "Indeed the notion of economic knowledge as highly developed schemata 
(interronnected cognitive structures) suggests a redefinition of economic knowledge as inextricably 
intertwined in a network that includes the linkages among bits of economic knowledge and the specialized 
procedures for using that knowledge" (Miller and VanFossen, 1994). 
Expertise and Schema 

In studies in fields as diverse as radiology, physics, and, political science, differences in how experts 
solve problems have been shown to depend upon more than knowledge of the content of their respective 
disciplines. To take a specific example, Lcsgold, et al. (1981) found that experienced radiologists were nearly 
always more successful in their diagnoses than recent interns, despite nearly equal content knowledge. Van 
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Sickle (1992) has noted that experts possess more declarative, procedural, and schematic knowledge than 
non-experts.^ Of these, only declarative knowledge refers to the knowledge of the specific body of content. 
Procedural knowledge is "the knowledge of how to" (Voss, 1989). Experts' more highly developed schemata 
point to greater integration among procedural and declarative knowledge. 

Miller and VanFossen (1994) summarized ten attributes of expert problem solvers drawn from the 
research literature. In a recent study, in which they developed a model to assess differences among experts 
and novices in economics. Miller and VanFossen found evidence that these attributes exist in the domain of 
econon-dcs. The attributes they listed (See Figure 1) included differences in declarative and procedural 
knowledge as well as more highly developed specialized schema. 

1. Experts excel mainly in their domain. 

2. Experts perceive relevant patterns in their domains. These meaningful patterns assist in the 
application of domain-specific knowledge. 

3. Experts see and represent problems at a deeper, more principled level than do novices. 

4. Experts spend more time on problem representation. Experts employ a 'work forward' 
strategy that requires greater time allocation for problem identification before the 
application of theory or knowledge. 

5. Experts have strong self-monitoring and self-evaluation skills. 

6. Experts demonstrate more flexibility in the process of problem solving. 

7. Experts identify models as tools in problem solving rather than real-life processes. 

8. Experts possess more domain-specific, declarative knowledge. 

9. Experts have extensive procedural knowledge. 

10. Experts have more highly developed specialized schetrmta than novices. 
Figure 1. Attributes of Expert Problem Solvers (Miller and VanFossen, 1994) 



^ Miller and VanFossen have pointed out that the term "schematic knowledge" is potentially confusing 
since it implies that it is a component of domain specific knowledge that is somehow separate from 
declarative and procedural knowledge. They argue that it is perhaps better to conceive of this as more 
highly developed domain specific schemata, the integration of declarative and procedural knowledge. 
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Expertise and Economic Education 

Th;i,t there should be evidence that economic expertise depends on more than just knowledge of 
economic concepts has been implicit in much of the activity of economic educators over the years. For 
instance. Buckles has written: "Our teaching of economics can be most effective if we focus on the ideas that 
economics is a 'method' an 'apparatus' and a 'technique'" (1991, p. 24). This statement implicitly assumes the 
existence of something like procedural knowledge. Much of the writing about the need or rationale for 
economic education has discussed economics in similar terms. Moreover, some of the materials developed 
by economic educators distinctly emphasizes employing the "method" of economics, including, for example, 
Mini-Society (Kourilsky, 1983), Capstone: The Nation's High School Economics Course (Reinke, et al., 1989) and 
United States History: Eyes on the Economy (Schug, et al., 1993). 

If this has been the case, why has so little research been devoted to the acquisition of expertise in 
employing the "method" of economics? While some studies have attempted to highlight the differences 
between experts and novices, there have been none depicting the nature of relative expertise, or any other 
such points along the path to expert status. Moreover, and more importantly, no research has been 
conducted about the most effective means of instruction in developing these various levels of expertise in 
economic problem solving. 

If the goal of economic education is more effective economic problem solving, it is important to 
know how students become more effective problem solvers and what instructional strategies and 
curriculum materials produce greater expertise. 'Toward this end, it is crucial to know how well the 
patterns of economic reasoning conform to those of economic experts. Economic educators might then be 
better able to design curriculum and instruction that addresses the development of economic reasoning 
based on a firm research foundation" (Miller and VanFossen, 1994). This study is intended to be a first effort 
in illuminating the path to economic expertise. 
Models of the Nature and Construct of Relative Economic Expertise 

In the absence of research, one might conceive of any number of possible models that depict the 
nature and construct of relative expertise in economic problem solving, each with attendant implications for 
instruction and curriculum design (Sec Appendix B). For instance, one possibility is a simple continuum. 

er|c ^» 
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As students acquire more domain-specific declarative knowledge, they progress in regular increments 
toward greater expertise in economic problem solving. This model seems unlikely given research on schema 
theory (See Tomey-Purta, 1991^ for a summary of this line of study), but it coincides best with the much of 
the teaching and research in economic education. It implies that material designers and teachers can 
concentrate on teaching economic concepts and content, and that we can assess expertise by measuring how 
well students have grasped economic com >ts. 

A more complicated construct is that greater instruction in economics has little noticeable impact on 
expertise in problem solving until one or more threshold levels are reached. One might think of this as a 
"discontinuous continuum," wherein a student moves ahead on the declarative knowledge dimension of 
expertise without making much discernable progress in problem solving until reaching a threshold, where a 
"leap" in problem solving ability occurs. That such a leap is plausible is suggested by the knowledge 
integration aspect of schema theory (VanSickle, 1992; Toniey-Purta, 1991; VanSickle and Hoge, 1991) . The 
student suddenly "gets it" as disparate economic ideas and notions of procedure become incorporated into 
her or his schemata. This implies the need to study how this knowledge integration takes place and what 
instruction and materials best promote it. 

Another possibility is suggested by the attributes of expertise as drawn from existing cognitive 
research (Glaser and Chi, 1988; Chi, Glaser and Rces, 1982). In this model, students proceed-probably at 
different rates-down multiple dimensions of the attributes of expertise in problem solving. The third model 
in Appendix B shows one such representation that includes dimensions for problem representation and 
procedural and declarative knowledge— three of the ten attributes listed by Miller and VanFossen (1994). 
This model suggests that researchers should concentrate on understanding how each of the attributes is 
developed with an eye toward instruction that promotes their progress. 

Perhaps the path to expertise is curvilinear. As depicted in Appendix B, there might be a series of 
levels of expertise with some distribution of, for example, standardized test scores at each level. Greater 
expertise in problem solving might at first respond slowly to successive increments of declarative 
knowledge. Later, the path of expertise turns more sharply upward as students build more highly 
developed schemata. While the depiction in Appendix B shows this possibility as a curve concave to the x- 
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axis, a curve convex to the ;';ds is also possible. The latter implies rapid progress in the acquisition of 
expertise with diminishing returns setting in at some point. The former implies that there might need to be 
substantial knowledge in economics before there is much not'ceable improvement in economic problem 
solving. Perhaps researchers would then focus on techniques ',o change the shape of the curve. 
Statement of the Problem 

Obviously, there are other possible models. The significant point for this study is that while we 
know something of the differences between expert and novice problem solving in economics (Miller and 
VanFossen, 1994; Son and VanSickle, 1993; VanSickle, 1992), we know almost notliing of how expertise is 
acquired and the relationship of economic education to acquisition of expertise. This study is intended as a 
first step in gaining insight into the nature and construct of relative expertise in economic problem solving. 
One necessary and significant aspect of the study is the identification and analysis of indicators useful in 
assessing relative expertise. 

Method 

The researchers examined literal transcripts generated from the 'talk-aloud' protocols of twenty- 
eight participants responding to three economic problems. Each of the following sub-groups contained four 
participants: high school students who had not taken economics (HS NOECON), high school students who 
had (HS ECON), undergraduate non-majors (UND NOECON), undergraduate economic majors (UND 
ECON), graduate students in economics (GRAD), Ph.D. economists employed in public and private 
forecasting (PHD-FIELD) and academic Ph.D. economists (PHD-UNIVERSITY). 
Research Desig n 

The study employed a causal-comparative design. According to Fraenkel and Wallen (1990), causal- 
comparative designs attempt "to determine the cause or consequences of differences that already exist 
between groups or among individuals" (p. 305) and also "involve comparing known groups who have had 
different experiences to determine possible causes or consequences of group membership" (p. 15). Causal- 
comparative research employs the static group comparison design (Campbell and Stanley, 1963). For this 
study, the group difference variable was defined as expected level of relative expertise in economic problem 
solving. This expectation was indit ated by the level of formal economic education attained by a participant 
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(the variable PARTLEV). Figure 2 provides a schematic that represents the static group comparison research 
design used in the current study. 



Group Independent Variables 

I 13 relevant indicators of expertise, 

(group characteristics) 



(Groups II-VD 



Dependent Variable 

relative expertise in 
economic problem solving 



IIV 13 relevant indicators of expertise, 

(group characteristics) 



relative exp>ertise in 
economic problem solving 



Figure 2. Design of the Current Study 



The static group comparison design posed several significant problems with respect to questions of 
internal and external validity. Among these were: (1) lack of randomization, (2) inability to manipulate 
independent variables, (3) subject selection and (4) data collector bias (McCracken, 1991; Campbell and 
Stanley, 1963). 

The current study addressed the lack of randomization and subject selection issues by creating 
homogenous sub-sample grouping and by employing purposive sampling techniques. Fraenkel and Wallen 
(1990) noted that one v^ay to control extraneous variables, outside of random selection, v^as to "find, or 
restrict, one's comparison to groups that are homogenous on tliat variable" (p. 310). Therefore, the current 
study used an across group comj^arison based upon participants' expected level of expertise in economic 
problem solving as indicated by the level of formal economic education attained. Further, sub-sample group 
members v^ere relatively equivalent v^ith regard to level of formal economic education and economic 
experience attained. Moreover, as it v^as not a goal of this study to engage in an experimental design, the 
issue of manipulating an independent variable was unimportant. 
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Finally, the questio" of data collector bias in the analysis of transcribed responses was addressed by 
having a second rater exai -dne and code a random sample of the respondents' protocols. This rater was 
extremely familiar with this line of study and was trained using the coding rubrics outlined in this chapter. 
The results of the second rater's coding indicated a very high degree of correlation between the researcher's 
coding and that of the second rater (correl ition coefficient = .8341; p < .01). Tliese results may be interpreted 
as partial validation of the coding process employed by the researcher in this study. 
Selection of Participants 

Members of the seven sub-sample groups were identified, either directly, or indirectly, by the 
primary researcher. For example, the academic Ph.D. economists (PHD-UNIVERSITY) were approached 
based on suggestions from a member of the economics faculty at The Ohio State University (OSU) and were 
all members of the OSU economics faculty. 

A similar process was undertaken to secure the participation of the non-university Ph.D.'s (PHD- 
FIELD). A membership list of the Columbus Association of Business Economists (CABE) was used to 
identify participants. The four economists who participated were employed in either public or private 
economic forecasting or analysis. 

A list of second- and third-year OSU graduate students in economics (GRAD) was used to generate 
a mailing to which four OSU graduate students (three second-year and one third-year) responded and 
subsequently participated in the study. A list of undergraduate economics majors at OSU was used to 
produce a mailing, out of which four participants were identified. Two of the participants wt-re of junior 
standing, one was a sophomore, and one was of senior standing. 

The undergraduate, non-economic maj^ir participants were members of a social studios education 
methods course taught by one of the researchers. Three of the four participants had never taken a formal 
economics course (high school nor university). The fourth participant had taken, as part of a Masters' 
Degree program in sc>cial studios education, a course titled Curriculum and Instruction in Global Economics, 
but no other formal economics courses. 

The four hiph school economics students were members of a senior-level economics class that the 
primary researcher observed during winter quarter, 1992. The four were selected by the researchers, in 
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conjunction with the student's classroom teacher, on the basis of their performance throughout the one- 
semester course. The students had just completed the economics course when the data were collected. It 
should be noted that this course was taught by a veteran high school teacher who had attended several 
workshops on teaching high school economics and whose course was based upon Ihe National Council on 
Economic Education's Framework for Teaching the Basic Concepts (Saunders, et al., 1993), considered by 
many to be the major economics curriculum in use today. 

This high school economics teacher also selected the four student participants who had not taken 
high school economics. These four were also seniors and data on this sub-sample group was also collected 
at the end of the term. Thus, in a broad sense, the two high school student sub-groups \.ere relatively 
equivalent except in formal economic education and economic experience. 
Data Collection 

Data collected and analyzed for the current study took the form of tranioibed participant responses 
to three pre-determined economic problems. These responses, or protocols, were audio-taped and then 
literally transcribed by the primary researcher. This data collection strategy was well supported by much of 
the previous expert problem solving research in cognitive psychology (See, for example, Chi and Glaser, 
1980; Glaser and Chi, 1982; and Lesgold, k 1981 and Voss, et al., 1983, 1989). The so-called 'talk-aloud' 
strategy used in the study asked participants, to the best of their ability, to verbally express their thought 
processes during the problem solving activity. That is, as participants dealt with various domain-specific 
problems, they were encouraged to "put into words" the processes they engaged in while addressing these 
problems. 

Participants were given a set of standardized instructions (See Appendix B) that allowed one minute 
for the respondent to familiarize themselves with each problem. The instructions explicitly encouraged 
respondents to draw any diagrams or graphs that might help them in dealing with the problem in question. 
No pre-set time limit was suggested by the instructions. Rather, the primary researcher stated to each 
participant that they should begin when ready and continue speaking until they had, to their own 
satisfaction, dealt sufficiently with the problem. Participants were told In continue analyzing the problem 
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until they felt certain they had exhausted their input and felt connfortable with their response. This process 
was repeated for each subsequent problem. 

Three economic problems—based upon those constructed by Miller and VanFossen (1994)~were 
employed in the study. Each oi these represented one of three broad areas of economic theory: 
microeconomics, macroeconomics and international trade. As Miller and VanFossen noted, these problems 
"were crafted to allow the researchers to detect important differences in economic problem solving that 
might be specific to the individual problems" (p. 15). Further, Miller and VanFossen believed, some issues 
of expertise in problem solving may be domain-specific or idiosyncratic. Thus, it was essential to develop a 
series of economic problems that would generate the broadest range of responses and therefore denrtonstrate 
the widest range of expertise with respect to economic problem solving. The problems developed were: 

1. Suppose Congress were to double the current minimum wage of $4.25 an hour to $8.50 an hour. 
Analyze the economic impact of this jX)licy and discuss whether you believe such a policy 
would be a good idea or not and why (miaoeconomic focus). 

2. In 1929, the so-called Great Depression began. Discuss what you believe caused the Great 
Depression and what, if anything, the federal government should have done to keep economic 
conditions from deteriorating so badly (macroecononiic focus). 

3. Trade among nations is a perennial economic issue. Suppose that you are the recently appointed 
Secretary of Commerce, and assume that our trade deficit has been growing (the US has been 
buying more goods and services from foreign countries than they have been buying from the 
US). As the Secretary of Commerce, your problem is to design and defend the new 
administration's trade policy. How will you respond (international trade focus)? 

Each respondent's transcribed protocol was analyzed by the primary researcher. The first phase of 
analysis involved the coding of protocols for eight variables: absolute number of relevant statements 
(STATE), percentage of relevant statements (RELPER), number of economic concepts (CONCEPTS), number 
of economic models (MODELS), number of concept maps (CONMAPS) ar.d expi<rt ratio profiles (ERP) for 
causal, (CAUSERP) propositional (PROPERP) and problem representation statements PROBERP). 

Thest; eight variables, along with the five Pitt Problem Solving Coding System variables outlined 
below, constituted a set of relevant indicators of expertise in economic problem solving. These indicators 
were developed through a content analysis of key studies in expert-novice problem s<jlving (Sec, for 
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example, Voss, et al., 1989; Gaser and Chi, 1988 and Glaser, 1987) and from a pilot study conducted to 
develop a model for assessing expertise in economic problem solving (Miller and VanFossen, 1994). 

The number of relevant statements (STATE) made by a participant during a, response was counted. 
For the current study, a relevant statement was defined as one complete sentence in a respondent's protocol 
that contained relevant economic information or that specifically addressed the problem under 
consideration. Similarly, the number of non-relevant statements were also counted. A ratio of relevant-to- 
non-relevant statements was then calculated (RELPER). 

The number of economic concepts (CONCEPTS) used by a respondent within a protocol were 
counted. For the current study, an economic concept was defined as a class of economic phenomenon that 
possessed common characteristics and/or attributes and that also noted linkages to other, broader more 
inclusive economic concepts. Moreover, for the sake of the current study, economic concepts were deemed 
those concepts whose relation to economic theory was generally accepted, or were considered low-inference 
concepts. A list of the economic concepts employed by participants in the current study may be found at 
Appendix C. 

Data were collected regarding the total number of economic models (MODELS) used in a 
participant's protocol. The primary researcher coded and counted the number of times a participant used 
various economic models during a response. During this data collection, an economic model was 
considered to be a complex series of conceptual connections, assumptions and rules for rendering economic 
interaction within a specific area of the domain of economics. Examples of such models include: general 
supply and demand models, aggregate demand and aggregate supply models, exchange rate markets, 
public choice models and industrial organization (I-O) models. A complete list of the economic models 
employed by participants in response to the three econonruc problems is located at Appendix D. 

The primary researcher collected data on the use of economic concept maps (CONMAPS) by first 
identifying the economic concepts used in a protocol by a respondent. A conceptual network was defir,ed as 
an interrelation among economic concepts such that the invoking of one economic concept led to the 
invoking of one or more other economic concepts. In addition, it was assumed that the use of a concept map 
by the respondent was analogous to the invoking of a specific cognitive structure—much like schemata. 
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Therefore, the concept maps took on different levels of complexity as they were associated with the 

explication of a particular economic model. For example, a discussion of the concept of supply and demand 

required a discussion of price and therefore equilibrium price, and so forth. However, this was a general 

market model. A second concept map might involve linking such a general model with a specific example— 

a labor market-and therefore price with real wage rates. 

Given this, the researchers coded an economic concept map as the following: a low-inference 

connection between two or more economic concepts situated somewhat contiguously within the context of a 

protocol and used in the explication of an economic model. Stated another way, if the respondent connected 

(or invoked) one or more economic concepts shortly after invoking a prior concept, and within the 

framework of an economic model, and this connection was essentially correct economically, the conceptual 

string created by the respondent was coded as a concept map. The following excerpt from a respondent's 

protocol is an example of such a concept map: 

"So this is short run, very short run, it is going to happen overnight. The there will a gap in quantity of 
labor demanded and quantity of labor supplied with unemployment resulting..." 

For this respondent, the use of the concept short run was followed closely by the invoking of three 
other, related concepts. These concepts were related in the sense that, economically, the concept of short run 
implies no time for markets to adjust to changes in factor inputs or other shocks and, therefore, the issue of 
"gaps in quantity of labor demanded..." occur only in the short run. Moreover, this map is directly related to 
the application of a generalized market model. This case is a more sophisticated example, as the issue of 
time, and its relationship to the market, is implied. Furthermore, this concept map is directly related to the 
first concept employed by the respondent: short run. 

Data concerning respondent's problem representation statement, causal statement and propositional 
statement Expert Ratio Profiles (ERP's) were coded using criteria previously employed by Miller and 
VanFossen (1994). Statements were classified as problem representation (PROBER?) if they demonstrated 
an attempt to re-order or re-construct a problem in an effort to discover connections, or to sort the problem 
into more accessible algorithms. Further, problem representation statements demonstrated some level of 
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planning on the part of a respondent with respect to the problem and processes used to address the problem. 

The following excerpt from a respondent's protocol illustrates a problem representation statement: 

"...if you're living in a world of fixed exchange rates, that sets up one set of problems. If you're living 
in a world with flexible exchange rates that sets up a different set of problems." 

Similarly, statements were coded as propositional (PROPER?) If they contained 
"links.. .which...resembled the 'if part of an 'if.. .then' statemeni" (Miller and VanFossen, 1994, p. 17). Thus, 
statements that represented some level of logical connection between an economic condition and an 
economic outcome were coded as propositional. The example used above to illustrate problem 
representation is also an example of a propositional statement. A second example of such a prepositional 
statement is "if you double the minimum wage, that means that more people will not be employed." 

Finally, statements that clearly established links of causality (CAUSERP) were coded as causal 
statements. These causal links were defined as statements that made an 'A causes B' distinction. Although 
similar in nature to the propositional statement, the causal statement involved the demonstration of a more 
direct economic connection rather than a hj'pothesis for examination. For example, a respondent stated that 
the income tax increase of 1932 led to a fall in disposable income and therefore a drop in aggregate demand. 
One can classify such a statement as fitting the 'A causes B' model. 

The second phase of analysis undertaken by the researcher involved coding the transcribed 
protocols in terms of the Pitt Problem Solving Coding System (PPSCS). First developed by Pitt (1983), the 
PPSCS (See Figure 2) coded qualitative data, such as the current study's transcripts, into one of six categories 
of strategies used in problem solving (general problem solving, feedback, pattern extraction, hypothetico- 
deductive, evaluation and heuristics) by integrating constant comparison analysis (Glaser and Strauss, 1967). 
Pitt (1983) argued that the coding system developed in her study provided a "comprehensive, empirical 
instrument to code heterogeneous verbal protocols in terms of the types of processing function each verbal 
proposition represents" (p. 551). 

Pitt (1983) provided operational definitions for each of the six coding classifications in the PPSCS. 
For the purposes of the current study, general problem solving strategies included defining an initial state of 
the problem, defining the goals involved, and being able to identify necessary data needed in solving a 



General Problem Solving 
SRIO. Define initial stale 
SRll. Define goal state 
SRIZ Identify data needed 

Feedback 
SRI 7. IdenHfy feedback 
SR18 Tag new information 
SR19 Organize new information 

Pattern Extraction 
SR22. Extract patterns from data 
SR23. Summarize relevant patterns 

Hypothctico-deductive 
SR 7. Formulate hypothesis 
SR 8. Etefine predictions 
SR20. Match data to predictions 
SR21. Determine truth values of predictions 



14 

Evaluation 

SR 2. List assumptions 

SR 4. Select evaluative criteria 

SR 5. Assign priorities 

SR 6. List relevant information 

SR15. Edit algorithm 

Basic Heuristics 
SR 1. List given information 
SR 3. List questions 
SR 9. Select questions 
SR13. Identify set of available algorithms 
SR14. Select algorithms 
SRI 6. Execute algorithm 
SR24. Output conclusions 



Figure 2. Components of the PPSCS Model: Problem Solving Strategies and Subroutines (SR)(Pitt, 1983) 



problem. Feedback strategies identified and incorporated new information as it became available during the 
problem solving process. Pattern extraction strategies referred to the identification of relevant patterns, 
symmetries or regularities in the assembled data. Hypothelico-deductive strategies involved the formulation 
of hypotheses, engaging in predictions and analyzing the validity of these predictions. Evaluation strategies 
suggested that problem solvers select evaluative criteria, assign priorities, and revise the problem solving 
strategy based upon the evaluation. Finally, Pitt identified a sixth category: basic heuristics. Basic heuristic 
strategies represented an abbreviated hcurist.c that can suffice for simple, familiar problems. This 
classification of strategies is more complex than simple trial-and error, yet is too simplistic for more complex 
problems. For purposes of the current study, however, data were coded on only the first five categories as 
the researchers believed that the basic heuristics category generated data that was very similar to the ERP 
data already calculated for each participant. 

Using the coding classifications outlined briefly above, and the constant comparison analysis 
technique, the researcher coded each protocol using the PPSCS. Mean levels for each PPSCS variable were 
then calculated for each of the seven sub-sample groups. 
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Data Analysis 

The study employed the use of inferential statistical analysis techniques in spite of the very small 

sample size that would appear to violate basic assumptions necessary for the use of these techniques. 

However, Kerlinger (1973) concluded that: 

"(Ulnless there is good evidence to believe that the populations are rather seriously non-normal, and that 
variances are heterogeneous, it is usui,lly unwise to use a non-parametric statistical test in place of a 
parametric one. The reason for this is chat parametric tests are almost always more powerful than non- 
parametric tests" (p. 287). 

Further, Kerlinger stressed, the F distribution is relatively insensitive to the shape of the distribution 
in the parent population and that unless the evidence is obvious that the variances are so heterogeneous as 
to be easily seen, the impact on the F test will probably be negligible. Kerlinger concluded that, even if the 
first two assumptions are violated, the f- and F-tests will be highly accurate. Harnett (1971) concluded that 
the "robust" nature of these distributions implied that their usefulness extended to conditions that do not 
meet the necessary basic assumptions (p. 173). 

In addition to this evidence, the study's use of a purposive sampling techniques added further 
support for the use of inferential statistical analysis. The researcher deliberately sought to identify members 
of the seven sub-sample groups for whom claims of representativeness might be easily defended. Indeed, 
close examine of one of these groups (the high school economics students) should provide the necessary 
insight in this case. 

The high school economics students had, as noted earlier, an a very well-trained high school 
economics instructor. This, coupled with the fact that the curriculum for the course was derived from the 
National Council on Economic Education curriculum, implied that the students' experience was certainly no 
worse than any other high school economics course, and perhaps even better than most. Indeed, it can be 
argued, that these students' experience was not easily recognizable as "seriously non-normal" in any sense. 

Therefore, the researchers employed inferential, parametric techniques in an effort to describe more 
completely the nature and constructs of relative expertise m economic problem solving. However, the 
researchers wish to acknowledge the assumptions that have not been met and to state explicitly that the use 
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of these techniques should not be construed as an attempt to broadly generalize to any population or group 
other than that described ai length within the context of the report. 

The first level of analysis conducted in this study involved the calculation of sub-sample group 
means and standard deviations for each of the 13 variables under investigation. Further, these means were 
compared using one-way analysis of variance (ANOVA) to determine if the means for each of the sub- 
sample groups differed significantly from the means of the other six sub-sample groups. The one-way 
application was selected because the respondents differed along only one factor of interest; namely, the level 
of expected expertise (the variable P ARTLEV). 

For the second level of analysis, the data on each of the thirteen variables were standardized for the 
sample by re-configuring the variable into absolute rankings. These rankings were used to calculate a mean 
ranking for each group across all variables. A correlation analysis was conducted on this rank data to 
determine if a relationship existed among any of the variables in question-including the rank order and 
mean ranking transformations-and the level of formal economic education of the participants (PARTLEV). 

Results 

Statistical Significance and the Current Study 

There is some debate concerning the issue of statistical significance versus practical significance in 
research in the social sciences and especially within education. McCloskey (1993) argued that '"statistically 
significant' does not mean 'substantively significant'" and that these two types of significance may have 
"nothing to do with each other" (p. 360). In this, McCloskey echoed Kruskal's (1968) line of argument that "it 
is easy to be over conservative and throw out an interesting baby with the nonsignificant bathwater...lack of 
statistical significance at a conventional level does not mean that no real effect is present" (p. 240). Indeed, 
Kruskal went on to note the "statistical significance of a sample bears no relationship to the possible subject 
matter significance" (p. 240). 

The researchers were fully aware that significance tests are attempts to determine whether a sample 
is large enough to ensure approximately equivalent results from subsequent samples. As such, these tests 
are, in effect, direct functions of sample size (take, for example, the f-statistic, where / is determined using 
the standard error of the mean which is a direct function of «)• Thus, results that have not proven to be 
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statistically significant for the icUowing analysis might plausibly result in significant results given a larger 
sample. 

The researchers believed that the study generated some intriguing results that did not meet the 
stringent requirements for statistical significance, but were nonetheless essential for developing a deeper 
understanding of the nature and constructs of relative expertise in economic problem solving. Therefore, the 
current study reported results that were, in the judgment of the researchers, statistically significant, 
practically significant, or both. 

The Nature and Constructs of Relative Expertise in Economic Problem Solving 

The research question investigated by the current study was the following: Based on a sample range 
of econorruc problem solvers, what is the nature and construct of relative expertise in economic problem 
solving? By comparing sub-sample group means and standardized rankings, it was possible to state 
explicitly what one sub-sample group looked like relative to the other sub-sample groups. Therefore, the 
purpose of this line of analyses was to provide a description of how sub-sample groups differed across the 
thirteen variables, identified as relevant indicators of expertise, relative to other sub-sample groups. 

For purposes of discussion, high school students with no formal economic education are Group 1 (or 
HS NONECON), high school economics students Group 2 (or HS ECON), undergraduate students Group 3 
(or UND NOECON), undergraduate economics majors Group 4 (or UND ECON), graduate students in 
economics Group 5 (or GRAD), Ph.D.'s not affiliated with OSU Group 6 (or PHD-FIELD) and OSU Ph.D. 
economists are Group 7 (or PHD-UNIVERSITY). 
Relevant Indicators of Expertise in Economic Problem Solving 

Number of Relevant Statements (STATE). The mean number and standard deviations of relevant 
statements used by respondents, according to level of formal economic education and experience, is 
presented in Table 1. The data presented here, and for all subsequent variables, reflect the sub-sample group 
means across all the three economic problems. The means reported here were calculated for each of the 
seven sub-sample groups and not for each of the three economic problems within each sub-sample group. A 
decision was made to collapse the data in this manner after a one-way analysis of variance indicated no 
significant difference in the means of sub-sample groups when compared across problems. That is, there 
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was no significant differenc? among the means for each of the variables considered, and across each of the 
sub-sample groups, for the three problems used. Therefore, unless otherwise noted, discussion of the data 
analysis in this section refers to combined means from each of the three economic problems. 

The data indicated a difference in number of relevant statements made for each of the seven sub- 
samples groups. A one-way analysis of variance (all ANOVA results are summarized in Table 1) indicated 
that the Ph.D.. economists (PHD-OSU, PHD-FIELD) made significantly more statements than the first four 
groups (HS-NO ECON, HS-ECON, UND-NO ECON, UND-ECON), and non-university Ph.D.'s made 
significantly more statements than the first five groups [F(6,77) = 13.6, p=.05]. For this sample, no significant 
difference existed between the mean number of statements made by high school economics students and 
high school students who had not taken economics. Further, no significant differences existed in the number 
of statements made by high school economics students, undergraduate students and undergraduate majors 
in economics. 

While the results were not all statistically significant, it was nonetheless interesting to note that both 
groups of Ph.D. economists used nearly twice as many relevant statements as the graduate students and 
three times the number made by the undergraduate majors. The difference between the number of 
statements made between both high school groups was marginal. Similarly, little difference existed between 
the undergraduate economics majors and non-majors. These results implied the number of statements made 
by the high school and undergraduate economics students were only marginally greater than counterpart 
groups. The data also indicated a substantial positive relationship (correlation coefficient =.68; p<.01. Sec 
Table 2) between a participant's level of formal economic education and the number of relevant statements 
used to respond to one of the three economic problems. 

Percentage of Relevant Statements Used (RELPER). The mean percentages, out of the absolute number 
of statements used, of relevant statements employed by respondents according to level of formal economic 
education and experience is presented in Table 1. A one-way analysis of variance suggested that the Ph.D. 
economists (Groups 6 and 7), the graduate students. (Group 5), the undergraduate majors in economics 
(Group 4) and the undergraduate non-majors (Group 3) used a significantly greater percentage 
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Table 1 

Summary Statistics and ANOV A Results by Sub-sample Group 
Group 

HS NOECON HS ECON UND NOECON UND ECON GRAD PHD-FIELD PHD-UNIVER. 

Variable 



STATE 



II* 




A73b 


0,00 




1 iifib 


77 flfi'^ 

£./ .KJO 


24.67*'* 


SD 




3.67 


3.83 


2.10 


4.60 


16.88 


13.77 




















1083 


3.1 6^ 


7.673^ 


7.923b 


21.75bc 


37.83<: 


30.33<: 


SD 


1.24 


1.85 


5.31 


2.23 


7.81 


26.80 


17.83 


RELPER 


















.55 


.71^ 


79ab 


.90°'^ 


94bc 


.93bc 


.96^^ 


SD 


.17 


.20 


.13 


.11 


.07 


.05 


.04 


MODELS 


















.42^ 


.67^ 


1.333^. 


2.503b 


3.92° 


7.08"'^ 


9.17*^ 


SD 


.52 


.78 


1.49 


.67 


1.24 


3.92 


4.74 


CONMAPS 


















.00^ 


.173 


.923' 


1 .253b 


3.75b 


6.92^^ 


7.50*^ 


SD 


.00 


.39 


1.16 


.97 


1.86 


4.68 


3.71 


nn y~*T^ i~*n n 

PROBERP 


















.053" 


.01 


.083b 


.15"'^ 


.23^^ 


.25^ 




SD 


.11 


.02 


.09 


.03 


1.07 


.07 


.09 


CAUShRP 


















.02^ 


.02^ 


.063" 


.073" 


.13^"^ 


19^^ 


.2r 


SD 


•;06 


.06 


.09 


.07 


.07 


.10 


.10 


PROPERP 


















.02^ 


.053 


.023 


.033 


.093 


.083 


.113 


SD 


.06 


.08 


.06 


.05 


.08 


.05 


.08 


rl 1 IGl-^ 


















.253 


.333 


.583 


1.003 


2.50" 


2.83" 


3.50" 


SD 


.45 


.65 


.51 


.43 


.52 


1.12 


1.62 


PITTHD 


















.17a 


.33a 


.753 


1.253 


3.33^" 


3.83^ 


3.17^ 


SD 


.39 


.49 


.75 


.75 


1.07 


1.80 


1.52 


PITTEVAL 


















.17a 


.50^ 


.583 


2.42^" 


2.59^^ 


3.25^ 


3.42^" 


SD 


.39 


.68 


.67 


.99 


.99 


.% 


1.08 


PITTPAT 


















.17=^ 


.50^ 


.503 


2.25bc 


2.08^^ 


3.25 


3.25^"^ 


SD 


.39 


6.7 


.67 


.86 


.99 


.75 


1.29 


PITTFEED 


















.25a 


.42^ 


.833 


2.58^ 


2.83^ 


3.42^ 


3.50^" 


SD 


.45 


.67 


.57 


.90 


1.03 


1.31 


1.09 



*Means with same superscript do not differ significantly at .05 level. 
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Table 2 



Correlation Coefficients: All Variables by Level of Participant 



ComUtion witk 
PARTLEV 



STATE 



RELPER 

CONCEPTS 

CONMAPS 

MODELS 

PROPERP 

PROBERP 

CAUSERP 

PITTGPS 

PI TTEV AL 

PITTFEED- 

PITTHD 

PITTPAT 



.S7SJ" 

.7121 

.S7(n** 

.7390" 

.7S88** 

3663" 

.S900" 

.6£\9" 

.79S8" 

.8087" 

.8020** 

.7464»* 

.7945** 



• - Slgnlf. LE .05 



Signif. LE .01 (Z-Uiled) 



of relevant statements than both groups of high school students [F (6,77) = 18.6, p=.05]. In fact, while the 
results were not statistically significant, it was also interesting to note that the percent of relevant statements 
used was greater than 90% for the undergraduate majors, graduate students and both groups of Ph.D.'s. 
Moreover, there was a large difference-nearly seventeen percentage points-in the percentage of relevant 
statements used by high school economics students relative to high school students who had taken no 
economics. These results indicated that, for this sample of economic problem solvers, those who had taken 
high school economics demonstrated quite a large difference in the percentage of relevant statements used 
relative to those who had not, while those who had taken advanced graduate studies in economics (PHD- 
FIELD, PHD-UNIVERSITY, GRAD) demonstrated only a marginal increase over their counterpart groups. 
The data indicated a very strong positive relationship (correlation coefficient =.71; p<.01. See Table 2) 
between a participant's level of formal economic education and the percentage of relevant statements used to 
respond to the three economic problems. Thus, for this sample of economic problem solvers, the data 
indicated that participants with greater levels of expected expertise, as indicated by level of fornrwl economic 
education, were more likely to use a greater percentage of relevant economic statements in their responses. 

Number of Economic Co}icq)ts Used (CONCEPTS). The mean number of economic concepts used by 
respondents, according to level of formal economic education and experience, is presented in Table 1. A 
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one-way analysis of variance indicated that the Ph.D. economists (Groups 6 and 7) used a significantly 
greater number of economic concepts than both groups of high school students and both groups of 
undergraduate students (f (6,77) = 14.9, p=.05]. Additionally, the graduate students used a significantly 
greater number of economic concepts than both groups of high school students. However, while the 
number of concepts used by graduate students was significantly greater than both groups of high school 
students, no significant difference was found between the number of concepts used by graduate students 
and the number used by undergraduate majors and non-majors, or between the two high school groups. 

Again, whi'e the results were not all statistically significant, it is important to note that the graduate 
students, university-Ph.D.'s and field based-Ph.D.'s used nearly three, four and five times the number of 
economic concepts respectively than did the two undergraduate groups. Further, there was ext.^mely little 
difference between the number of concepts used by both undergraduate majors and non-majors and 
marginal differences between the two high school groups. These data suggested that those who had 
graduate training (GRAD), and advanced graduate training in economics (PHD-FIELD, PHD-OSU) 
demonstrated a much greater use of economic concepts relative to the other four groups, while those who 
had taken high school and undergraduate economics demonstrated little or no difference in the number of 
concepts used over counterpart sub-sample groups. These data also indicated a substantial positive 
relationship (correlation coefficient -.67; p<.01. See Table 2) between a participant's level of expected 
expertise, as indicat 1 by formal economic education and experience, and the number of economic concepts 
used in response to the three economic problems. 

Number of Concept Maps Used (CONMAPS). The mean number of economic concept maps used by 
respondents, according to level of formal economic education and experience, is presented in Table 1. A 
one-way analysis of variance suggested that the Ph.D. economists (Groups 6 and 7) used a significantly 
greater number of economic :oncept maps than both groups uf high school students, both groups of 
undergraduate students, and the graduate students in economics [F (6,77) = 20.4, p=.051. In addition, the 
graduate students used a significantly greater number of economic concept maps than both groups of high 
school students. However, while the number of concept maps used by graduate students was Jgnificantly 
greater than both groups of high school students, no significant difference was found between the number of 
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concepts maps used by graduate students and undergraduate majors and non-majors or between the two 
high school groups. 

In fact, while not all ol the results were statistically significant, it was noteworthy that both Ph.D. 
groups used approximately twice the number of concept maps as the graduate students. Moreover, the high 
school economics students and the undergraduate majors in economics used only marginally greater 
numbers of concept maps than counterpart groups, while the graduate students used exactly three times the 
number of concept nwps used by the undergraduate majors. These results suggested that, for this sample of 
economic problem solvers, those who had graduate, and advanced graduate, training in economics 
demonstrated greater usage of economic concept maps. Conversely, those who had high school or 
undergraduate economics demonstrated only a marginally greater use of economic concept maps over their 
respective counterpart groups. These data indicated a very strong positive relationship (correlation 
coefficient = .73; p< .01, See Table 2) between a participant's level of formal economic education and 
experience and the number of economic concept maps used in respxinse to the three economic problems. 

Number of Economic Models Used (MODELS). The mean number of economic models used by 
respondents, according to level of formal economic education and experience, is presented in Table 1. A 
one-way analysis of variance indicated that the Ph.D. economists (Groups 6 and 7) used a significantly 
greater number of economic models than both groups of high school students and both groups of 
undergraduate students, and that the university Ph.D.'s used significantly more economic models than the 
graduate students in economics [F (6,77) = 22.3, p<.05]. In addition, the graduate students used a 
significantly greater number of economic models than both groups of high school students. While the 
number of concept maps used by graduate students was significantly greater than both groups of high 
school students, no significant difference was found among number of concepts used by graduate students 
and those used by undergraduate majors and non-majors in economics or between the two high school 
groups. 

Again, while not ail statistically significant, these results are of some practical significance. It is 
particularlyintcrcsting to note that the universlty-Ph.D.'s and field based-Ph.D.'s invoked nearly two and 
three times, respectively, more economic models during responses than did the graduate students in 
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economics. Further, it is important to stress that, once again, those who had participated in high school 
economics demonstrated only very small differences over the counterpart group. However, for this sample 
of economic problem solvers, undergraduate economics majors used nearly twice as many economics 
models as did undergraduate non-majors. These results suggested that those with undergraduate, graduate 
and advanced graduate training in economics employed a significantly greater number of economic models. 
Moreover, those with training in high school economics displayed very little difference in the number of 
models used relative to the counterpart sub-sample group. These data also indicated a very strong positive 
relationship (correlation coefficient - .76; p<.01, See Table 2) between a participant's level of expected 
expertise, as indicated by formal economic education and experience, and the number of economic models 
used in response to the three economic problems. 

Problem Representation Expert Ratio Profile (PROBERP). The problem representation ERF was 
calculated by dividing the number of statements used by the respondent to reorder, reclassif\' or subdivide 
the problem by the total number of relevant statements used. Thus, this ERF was a ratio of problem 
representation statements used to total relevant • atements used. The mean problem representation ERF of 
respondents, according to level of formal economic education and experience, is presented in Table 1. A 
one-way analysis of variance implied that the Fh.D. economists (Groups 6 and 7), the graduate students 
(Group 5) and the undergraduate majors in economics (Group 4) had significantly higher problem 
representation ERF's than both high school groups and the undergraduate non-majors IF (6,77) - 16.2, 
p<.05]. Similarly, the graduate students in economics had significantly higher problem representation ERF's 
than did the undergraduate non .\ajors and both high school groups. 

While not all statistically significant, these results are nonetheless interesting. For example, the 
graduate students and both Ph.D. sub-groups used nearly the same percentage of problem representation 
statements. This level was twice the percentage used by undergraduate economics students and nearly three 
times the percentage used by the other undergraduate group. Curiously, the high school students without 
economic training used a greater percentage of problem representation statements than the high school 
economics students. These findings suggested that the use of problem representation in problem solving, 
for this sample of economic problem solvers, is more closely associated with the experience of graduate and 
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advanced graduate training in economics. However, the participants who experienced undergraduate 
training also demonstrated relatively frequent use of problem representation statements. The researcher 
was unable to develop a cogent argument for the relatively low levels of problem representation 
demonstrated by Group 2, although one supposition might involve a kind of "fishing around" scenario. That 
is, as one achieves a certain level of expertise, it may be the case that attendant levels of problem 
representation decrease at the margin—experts focus on the problem and narrow it down immediately. 
Those with less expertise may be forced to cast and re-cast a problem v/hile searching for the problem's 
significant elements, and thus use more PROBERP than expert counter-parts. 

These data indicated a substantial positive relationship (correlation coefficient = .68; p<.01. See Table 
2) between a participant's level of expected expertise, as indicated by formal economic education and 
experience, and the ratio of problem representation statements to total statements used. 

Causal Statement ERP (CAUSERP). The causal statement ERP was calculated by dividing the number 
of causal statements-those that resembled an 'A causes B' model-used by the respondent by the total 
number of relevant statements used. Thus, this ERP was a ratio of causal statements to total relevant 
statements. The mean causal statement ERP of respondents, according to level of formal economic 
education and experience, is presented in Table 1. The data indicated differences in each of the seven sub- 
sample groups' causal statement ERP. A one-way analysis of variance suggested that the Ph.D. economists 
(Groups 6 and 7), had significantly higher causal ERP's than both high school groups and both 
undergraduate groups [F (6,77) = 12.2, p<.051, and the graduate students in economics had significantly 
higher causal ERP's than did both high school groups. In fact, in absolute terms, no difference existed in the 
percentage of causal statements used by both high school groups. Additionally, the undergraduate 
economics majors used only marginally greater percentages of causal statements. Conversely, the Ph.D. 
groups used tremendously greater percentages of casual statements than did both high school groups and 
both undergraduate groups, and nearly seventy-five percent more than the graduate students. 

These data also suggested a substantial positive relationship (correlatioi coefficient = .69; p<.01. See 
Table 2) between a participant's level of formal economic education and experience and the ratio of causal 
statements to total statements used. 
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Propositional Statement ERF (PROPERP). The prepositional statement ERP was calculated by 
dividing the number of statentents that resembled an 'if.. ..then' model by the total number of relevant 
statements used. Thus, this ERP was a ratio of propositional statements to total relevant statements. The 
mean propositional statement ERP of respondents, according to level of formal econonnic education is 
presented at Table 1. A one-way "analysis of variance indicated that these differences were not significant [F 
(6,77) = 3.3, p<.051. 

Though the ANOVA indicated no significant differences in the percentage of propositional 
statements used, some commentary on these results is still warranted. For this variable, the high school 
economics students used nearly three times the percentage of propositional statements as did the 
counterpart group. Similarly, the undergraduate majors used nearly twice the percentage of propositional 
statements as did the undergraduate non-majors. Finally, the graduate students and both Ph.D. groups used 
nearly three times the percentage of causal statements of the undergraduate majors. These data also 
indicated a moderate positive relationship (correlation coefficient = .39; p<.01, b.?e Table 2) between a 
participant's level of formal economic education and experience and the ratio of propositional statements to 
total statements used. 

Figure 3 represents a graphical comparison of the first four variables described above: STATE, 
CONCEPTS, CONMAPS, and MODELS. Figure 4 represents a graphical comparison of the variables 
RELPER, PROBERP, PROPERP and CAUSERP. 

It was evident from both Figure 3 and Figure 4 that the level of most of these eight relevant indictors 
of expertise in economic problem solvint^ increased across level of economic education (PARTLEV). Indeed, 
this trend appeared to hold for each of the eight variables under consideration. However, in several cases 
(e.g., STATE, CONCEPTS) it was also evident that the pattern noted above did not always hold. 

A second important point to be gleaned from Figures 3 and 4 involve the size of increase in the 
levels of the eight variables across various levels of economic education. Indeed, the increase in the level of 
the eight variables acrosi. level of economic education was neither uniform nor, in most cases, continuous. 
These lesults will be discussed further in subsequent analysis in this report, but suffice to »ay. Figures 3 
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Figure 3. Comparison of Summary Statistics: STATE, CONCEPTS, CONMAPS, MODELS 
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Figure 4. Comparison of Summary Statistics: RELPER, PROBERP, PROPERP, CAUSERP 
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and 4 offer evidence that, for this groiip of problem solvers and for this set of variables, the acquisition of 
expertise in economic problem solving was not a continuous phenomenon with equi-distant categories. 
The Pitt Problem Solving Coding System (PPSCS)Variables 

As noted earlier, the PPSCS provided "a comprehensive, empirical instrument to code 
heterogeneous verbal protocols in terms of the type of processing function each verbal proposition 
represents" (Pitt, p. 551). In addition, the PPSCS provided a reasonable method for classifying verbal 
responses along a commonly accepted problem solving mcdel. Specifically, use of the PPSCS helped present 
a more detailed picture of the general economic problem solving abilities and strategies of the respondents. 

PPSCS: General Problem Solving (PmCPS). The general problem solving measure was generated by 
counting the number of statements coded as meeting the Pitt GPS criteria. The mean GPS of respondents, 
according to level of formal economic education and experience, is presented in Table 1. A one-way analysis 
of variance indicated that these differences were not significant [F (6,77) = 28.8, p<.05] among Groups 1-4. 
Additionally, the ANOVA indicated that Groups 5-7 did not differ significantly. However, Groups 5-7 did 
differ significantly from Groups 1-4. 

These data indicated that the graduate students, and both Ph.D. groups, used significantly more Pitt 
GPS statements than both high school groups and both undergraduate groups. The high school economics 
students used only a marginally greater number of Pitt GPS statements than the counterpart group, while 
the graduate students used two and one-half times more Pitt GPS statements than undergraduate majors. 
The university-Ph.D. group used three and one-half times more GPS statements than the graduate students 
and a marginally greater number than the field-based Ph.D.'s. These data also suggested a very strong, 
positive relationship between Pitt GPS statements used and formal level of economic education (correlation 
coefficient = .80, p < .01; See Table 2). 

PPSCS: Hyplhetico-deduclive. (PITTHD) The Pitt hypothetico-deductive (HD) measure was generated 
by counting the number of statements coded as meeting the Pitt HD criteria. The mean HD of respondents, 
according to level of formal economic education and experience, is presented in Tablo 1. A one-way analysis 
of variance indicated that these differences were not significant [F (6,77) = 28.8, p<.051 among Groups 1-4. 
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Additionally, the ANOVA indicated that Groups 5-7 did not differ significantly in the number of Pitt HD 
statements used. However, Groups 5-7 did differ significantly from Groups 1-4. 

Moreover, these data suggested that those participants who had experienced high school economics 
used twice the number of Pitt hypothetico-deductive statements as the counter part group. Despite this, the 
absolute difference in the mean number of Pitt HD statements used by both high school groups and the 
undergraduate non-economics majors was so small as to be relatively insignificant. Indeed, even the 
undergraduate majors used an average of only slightly more than one Pitt HD statement per response. 
However, both groups of Ph.D.'s, and the graduate students,used approximately three times the number of 
Pitt HD statements as the undergraduate majors. 

PPSCS: Pattern Extraction (PITTPAT). The Pitt pattern extraction measure was generated by 
counting the number of statements coded as meeting the Pitt PAT criteria. The mean PAT of respondents, 
according to level of formal economic education and experience, is presented in Table 1. A one-way analysis 
of variance indicated that these differences were not significant IF (6,77) = 29.18, p<.05] among Groups 1-3 or 
between Groups 4 and 5. Additionally, the ANOVA indicated that Groups 4 and Groups 6 and 7 did not 
differ significantly in the number of Pitt PAT statements used. However, Groups 4 and 5 did differ 
significantly from Groups 1-3, and Groups 6 and 7 differed significantly from Groups 1-3 and from Group 5. 

More importantly, however, these data suggested that those participants who had experienced high 
school economics used three times the number of Pitt extraction statements as participants who had not 
experienced high school economics. Interestingly, the mean number of Pitt pattern extraction statements 
used by high school economics students and the undergraduate non-economics majors was equivalent. 
Even more surprising was the fact that while the undergraduate majors used nearly five times the number of 
Pitt pattern extraction statements as the undergraduate non-majors, the graduate students used slightly 
fewer pattern extraction statements, on the' average, than the undergraduate majors. Finally, both groups of 
Ph.D. economists used the same mean number of pattern extraction statements. This mean represented a 
fifty percent increase in pattern extraction statements over the undergraduate majors, and a seventy-five 
percent increase over the graduate students. 
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PPSCS: Feedback (PlfTFEED). The Pitt feedback (FEED) measure was generated by counting the 
number of statements coded as kivjeting the Pitt FEED criteria. The mean number of feedback statements of 
respondents, according to level of formal economic education and experience, is presented in Table 1. A 
one-way analysis of variance indicated that these differences were not significant IF (6,77) = 29.71, p<.05] 
among Groups 1-3. Additionally, the ANOVA indicated that Groups 4-7 did not differ significantly in the 
number of Pitt FEED statements used but Groups 4-7 did differ significantly from Groups 1-3. 

These data indicated that while the expected pattern of increasingly greater usage of PITTFEED 
statements held for the sample groups, a substantial difference existed in this pattern of usage between 
Groups 1-3 and Groups 4-7. That is, the undergraduate economics majors (and groups above) used a much 
greater number of PITTFEED statements than any of the first three groups. The groups above the 
undergraduates majors used more PITTFEED statements, but only marginally so. Thus, the data suggested 
that participants who had experienced undergraduate economic training-and those with economic training 
beyond-"leaped" ahead of the previous three groups with respect to PITTFEED usage. 

PPSCS: Evaluation (PITTEVAL). The Pitt evaluation measure was generated by counting the 
number of statements coded as meeting the Pitt evaluation statement criteria. The mean number of 
evaluation statements of respondents,, according to level of formal economic education and experience, is 
presented in Table 1. A one-way analysis of variance indicated that these differences were not significant [F 
(6,77) = 31.39, p<.05l among Groups 1-3. Additionally, the ANOVA indicated that Groups 4-7 did not differ 
significantly in the number of PITTEVAL staterrients used. However, Groups 4-7 did differ significantly 
from Groups 1-3. 

Again, while not all the differences among the mean number of evaluation statements made were 
statistically significant, some interesting patterns continued to emerge. As with each of the other Pitt 
Problem Solving variables, participant's usage of PITTEVAL statements followed the expected pattern of 
increasingly greater usage as the level of formal economic education and experience of participants 
increased. More importantly, however, the data regarding the number of PITTEVAL statements used 
confirmed the presence of a second pattern. This pattern involved a "lcap"~a significant increase in level of 
usage of a Pitt variable over previous groups-of qualitatively different proportions. That is, at some level of 
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economic expertise, participants di-monstrated a substantially greater usage of the Pitt Problem Solving 
attributes than had groups with less economic education. For three of the Pitt variables (PITTEVAL, 
PITTFEED, and PITTPAT), this "leap" was manifest in the undergraduate economic majors. For the 
ren^aining Pitt variables (PITTHD and PITTGPS), this 'leap" was evident from the graduate students. What 
these results imply is that, for aspects of problem solving measured by the PPSCS, a clear pattern concerning 
the acquisition of expertise in economic problem solving emerged. Indeed, the data suggested that, at least 
for this small sample of economic problem solvers, certain attributes of formal training in economics led to a 
substantial rise in the level of usage of PPSCS variables. 

Figure 5 represents a graphical comparison of the sub-group means for each of the Pitt variables 
described above and provides evidence that the pattern noted for the first eight variables continued to hold. 
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That is, the levels of PPSCS variable demonstrated increased across level of economic education. What was 
even more interesting, for these PPSCS variables, was the significant change in the level of all PPSCS 
variables associated with either undergraduate or graduate training in economics. In fact, it should be noted 
that Figure 5 also seemed to indicate the presence of a "leap" in level of expert-like problem solving-for this 
sample of problem solvers~at some point along the path to expertise and thus provided further evidence for 
the apparent discontinuuity of expertise acquisition for this small sample. 
Standardized Rank Data 

The last set of results for this study involved data concerning the absolute rankings of participant 
responses across each of the variables under investigation. Using SPSS to re-configure the raw data set for 
each variable, a new series of variables was created: nr»ean absolute ranking of respondents by participation 
level. By comparing the ranking of all participant responses relative to the position of other respondents 
(and resolving all ties with the mean position of the tied responses), this analysis provided further insight 
into the nature and constructs of relative expertise by allowing the calculation of a series of mean rankings 
for each variable across participant level (PARTLEV). With these mean rankings (calculated on the range 1- 
84), it was possible to examine the position of each sub-sample group-across each variable— relative to the 
other sub-sample groups. A comparison of these mean rankings may be scan in Figure 6 and Figure 7. It 
should be noted that, across most of the variables studied, the mean rankings corresponded well with 
participant's level of economic education. In fact, a correlation analysis of the relationship between the mean 
rankings and participant's level of formal economic education ranged from moderate to very strong. Table 4 
reports these correlation coefficients. These results suggested that, at least for this sample of economic 
problem solvers, the initial supposition that the level of pjarticipant's formal economic education was related 
to expected levels of expertise was accurate. Indeed, the mean ranking data confirmed that, across each of 
the variables, respondents' performance on each variable tended to rank according to their level of economic 
education. 
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Mean RANK of CONCEPTS 

••O- Mean RANK of CONMAPS 

•O—- Mean RANK of MODELS 
-a--- Mean RANK of STATE 
-ffl- - - Mean RANK of PITTGPS 



Figure 6. Comparison of Mean Standardized Rankings of CONCEPTS, CONMAPS, MODELS, STATE and 
PITTGPS 




-O- 



- V- 
-H- 



Mean RANK of CAUSER? 
Mean RANK of PITTEV AL 
Mean RANK of PITTFEED 
Mean RANKofPnTHD 
Mean RANK of PITTPAT 
Mean RANK of PROBERP 
Mean RANK of PROPERP 
Mean RANKofRELPER 



Figure 7. Compaiison of the Mean Standardized Rankings of CAUSERP, PITTEVAL, PITTFEED, VITYHD 
and PIITPAT 
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Table 4 

Correlation Coefficients for Mean Standardized Rankings of All Variables with Level of Participants 



ULinklViriable 


Cueffldent 


RMODELS 


.8570** 


RCONCEPT 




RPITTGPS 


.«376" 


RCONMAPS 


^9" 


RPITTFEE 




RPITTEVA 




RPITTPAT 


.80S2" 


RPITTHD 


.7998" 


RSTATE 


.7937" 


RRELPER 


.7061 


RCAUSERF 


.6984" 


RPROBERF 


.6780** 


RPROPERP 


.4803** 



• - Signif. LE .05 " - Signif. LE .01 (2-talIed) 

However, it was also interesting to note the exceptions to these trends. For example, the mean 
rankings of the graduate student's problem representation ERP were above both Ph.D. groups. Similarly, 
the high school students with no economics ranked above the high economics students on the problem 
representation ERP (see Figure 7). As noted earlier, it may very well be the case that these results were due 
to some kind of "fishing around" on the part of respondents. As noted earlier, it may be the case that 
attendant levels of problem representation decrease at the margin. Thus, those with greater levels of 
expertise may be able to a problem with respect to sub-problem identification, while those with less 
expertise must continually re-cast a problem while searching for the problem's significant elements. 

However, these exceptions, in conjunction with the data concerning the mean rankings, seemed to 
refute the supposition that the path to expertise Vv^as a continuum, with equi-distant categories along it. 
Indeed, although it was true that the order of the mean rankings for respondents was relatively consistent 
with intuitive expectations based upon participant's level of formal economic education, the intervals in 
these mean rankings were by no means equally distributed across each variable. These results suggested 
that the process of moving from one category of expertise to another was not the same across all sub-group 
categories. Certainly this was evident in the fact that, for several sub-groups, the calculated mean ranking 
on certain variables was above or below generally expected levels. The case of the problem representation 
statements noted above was one such example. Other examples included the mean rankings of number of 
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relevant statements used, the mean rankings of prepositional statements made and the mean rankings of the 
number of Pitt evaluation statements made (See Figure 7). Moreover, if the path to expertise were 
continuous, the movement along such a continuum would imply intervals. The data suggested that, for the 
variables noted above, this was not the case. 

Finally, as the data were standardized by rank, it was possible to calculate a mean of the mean 
rankings across all variables (Table 4). The results of this analysis suggested that the intervals between mean 
rankings were not equi-distant relative to each other. Indeed, it was apparent that the mean rankings fell 
into three groupings with respect to the standardized data: the two high school groups, the two 
undergraduate groups and the graduate students and both Ph.D. groups. 



Table 4 

Mean of the Mean Standardized Rankings (1-84) for AU V iriables 

Ii-vfl nf Participant MEAN OF MEAN RANK 

1 16.31 

2 22.11 

3 3a78 

4 37.95 

5 59.75 

6 63.58 

7 64.39 



The ranked data were further analyzed across all thirteen variables using hierarchical cluster 
analysis to determine if underlying groupings were present. The cluster analysis technique compared the 
similarity or distance associated with a group of cases and classified like variables into groups, or clusters, 
based on likeness. The results of this cluster analysis revealed that the n>ean cluster level (based on degree 
of association relative to all other groups) for both groups of high school students was 1.04. This implied 
that a first cluster was created around the first twenty-four rankings. In other words, all but two of the first 
twenty-four cases were assigned to the first cluster~a very high degree of similarity. 
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The analysis grouped both undergraduate sub-groups into a second cluster and the mean cluster 
level for both groups was 1.74. This implied that a cluster-but perhaps not as tight a cluster as for the first 
twenty-four-was created around the second twenty-four rankings. Indeed, nineteen out of these twenty- 
four cases were assigned to the second cluster. For the graduate students and both Ph.D. groups, the mean 
cluster level was 2.94. This implied that a third cluster was created around the last thirty-six rankings. 
These results suggested that the three groupings outlined in the previous mean ranking analysis were 
plausible. What these results also seemd to confirm was the presence of a categorization of expertise beyond 
the simple expert/novice construct; namely a category of relative expertise in economic problem solving. 

Figure 8 represents a graphical depiction of the mean rankings across participant levels and these 
results suggested two important conclusions. First, it was evident from Figure 8 that, to some extent, the 
"leap" concept introduced earlier in this analysis was also present in this data. In fact, one should note the 
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Figure 8. Comparison of the Means of Standardized Mean Rankings for All Variables Across Level of 
Participant 
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obvious change in slope associated with movement from undergraduate major to graduate student. 
Similarly, there was also a certain flattening in the mean rankings associated with movement from graduate 
student to both level of Ph.D.'s. These results seemed to confirm suppositions made earlier in this report 
concerning the nature of expertise and its acquisition. Namely, that the path to expertise, at least for this 
small sample of economic problem solvers, was not continuous and that, at the margin, certain levels of 
economic education appeared to exhibit diminishing returns with resjxjct to increased acquisition of 
expertise in economic problem solving. 

Conclusions 

The current study has been an attempt to learn more about expertise in economic problem solving and the 

nature or relative expertise. Given the plethora of data, and the analytical techniques brought to bear upon 

them, it would be ruce to draw some clear-cut conclusions about problem solving in economics. This, 

however, is not the case. Based upon the Htany of limitations and assumptions associated with the study, 

one should only seek to draw the most guarded and tentative conclusions. The reader is also reminded that 

the application of many of the analytical techniques used in the current study was done to more accurately 

describe the sample in question and should not be generalized to larger populations without careful 

consideration and critique. 

Having stated these caveats then, what follows is a brief list of the outcomes of this study. 

1 . Indications of a very strong, positive relationship between each of the thirteen variables and the level 
of expected expertise in economic problem solving as indicated by participant level. 

2. Identification of mostly marginal differences in the observed mean levels the thirteen variables, 
indentified as indicators of expertise, for the high school economics students over tV >se who had not 
taken high school econonrvics. A sinrvilar pattern was present in the observe^! mean levels of variables 
for both Ph.D. groups over the graduate students. Finally, the data indicated that some significant 
differences did exist between undergraduate economics students and their non-major counterparts. 

3. Identification of a pattern in the observed level of Pitt Problem Solving variables across levels of 
expected expertise. This pattern involved a 'leap"-or a significant increase in usage of Pitt variables 
over previous groups-of qualitatively different proportions. At certain levels of expected econonrvic 
expertise, participants demonstrated a substantially greater usage of the Pitt Problem Solving 
attributes than had groups with less expected economic expertise. 
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4. Support for the pattern noted above in the results of the mean standardized ranking analysis. Indeed, 
a hierarchical cluster analysis suggested that the rankings group)ed into three clusters: (a) both high 
school groups; (b) both undergraduate groups; and (c) the graduate students and both Ph.D. groups. 

5. Support for the hypothesis that the path to expertise was not a continuum with equi-distant categories 
along it. This support lay in results from both the mean ranking analysis and the analysis of 
percentage change in observed levels of the variables across the categorical levels of participants. 

Beyond these guarded conclusions, there are some other highly speculative, yet intriguing 
possibilities along the path to achieving economic expertise. The focus in economic education has been 
on the teaching of economic concepts. The findings in this study show no differences of practical or 
statistical significance between either of the high school groups, nor between the two groups of college 
students. Moreover, the college students who had no formal economics courses, high school or college, 
invoked concepts in their responses more than twice as often as the high school studeiits who had taken 
economics. 

One possible conclusion is that the teaching of economic concepts has made little difference in the 
learning of these students. Alternatively, and the researchers think more likely, the students who have 
had formal economics instruction have little increased ability to use the concepts they have learned in 
responding to economic problems. The performance of Group 3 (UND-NOECON) suggests that they 
achieved their modest level of conceptual learning without the benefit of an economics course. But, they 
were as able to display what economic concepts they had acquired (presumably from life experiences), 
better than high school students and equally as well as there college counterparts. This might be 
interpreted as evidence for growth in their economics schema - the bits of economic knowledge are more 
connected and accessible, and therefore more useful in solving problems. 

When looking for evidence of other dimensions of expertise beyond declarative knowledge, the 
data suggest two especially significant points in the path to expertise. Despite the lack of difference 
between the college students on the concept variable, there are marked difference of practical significance 
on three of the Pitt variables (which were also statistically significant) and their use of economic models. 
One possible interpretation of these results supports the speculation above — that college economics 
begins to produce sonw integration of the knowledge students have into a more unified whole. 
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Perhaps this phenomenor is best seen in the MODELS variable, one of particular significance in 
economics since the discipline relies so heavily on the development and use of models. The 
undergraduate economics students in this study used models in their responses three and one-half times 
as often as the high school economics students, and nearly twice as often as their undergraduate 
counterparts who had no economics. The graduate students invoked models more than the 
undergraduates, and the university professors employed models more than twice as often as the graduate 
students. One might speculate that models are the basic integrating structure of the discipline and that 
their use is a significant characteristic of a developing economics schema. 

The second point, which has been mentioned already, is study of graduate level economics. As 
noted above, this emerges most clearly in Figure 8 comparing the means of the standardized mean 
rankings. This figure summarizes the results across all variables and points strongly to a sizable change 
in the expertise in economics of the members in this group, at least in this sample. 

It is unreasonable to expect economic educators to provide graduate level understanding to high 
school students. If economics problem solving is the real goal of economic education, however, then 
perhaps by focusing more clearly on otlier dimensions of expertise in high school economics, and by a 
clearer understanding of what distinguishes relative expertise, economic educators can help high school 
students become more adept economic problem solvers. 
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Appendix A: 

Relevant Indicators of Expertise in Economic Problem Solving (Variables 

Investigated) 



STATE 

RELPER 

CONCEPTS 

MODELS 

CONMAPS 

PROBERP 

PROPERP 
CAUSERP 
PITTGPS 
PITTHD 

PITTFEED 
PITTI^AT 

PITTEVAL 



Absolute Number of Relevant Statements Made 

Percentage of Relevant Statements 

Number of Economic Concepts Used 

Number of Economic Models Used 

Number of Concept Maps Used 

Problem Representation Statement to Absolute 
Statement Ratio 

Propositional Statement to Absolute Statement Ratio 

Causal Statement to Absolute Statement Ratio 

Pitt Coding Variable: General Problem Solving 

Pitt Coding Variable: Hypothetico-deductive 
statements used 

Pitt Coding Variable: Feedback statments used 

Pitt Coding Variable: Pattern extraction statements 
used 

Pitt Coding Variable: Evaluation statements used 
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APPENDIX B 



HYPOTHETICAL MODELS OF THE ACQUISTION OF EXPERTISE IN 
ECONOMIC PROBLEM SOLVING 



4S 



1. Continuum 



Novice 



Expert 



2. Discontinuous Continuum 



Novice 



3. Multiple Continua 



Expert 



Expert 



Contenl KnowWt* 

Novice 



4. Curvilinear* 



Expert 



Novice 



• TIk <lo<tc<l line in this ocn«nKi coonwn the mean* of a ran^ 
tndKatoo. JMOcuted a sub^pJe of the wudy and repre.eno one hypothetical 
•»hape' of expert oe. 
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APPENDIX C 

STANDARDIZED PARTICIPANT DIRECTION SHEET FOR TALK- ALOUD' 

PROTOCOLS 
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Directions for Economic Problems Exercise 



In a moment, you will be given the first of three economic problems. 
Once you have the problem sheet in front of you, please note the following 
directions: 

1. you will have one minute to familiarize yourself with the problem 

2. at the end of this time, please begin to discuss the problem: 

- speak clearly 

- be as detailed as you are able 

- take as long as is necessary to discuss the problem thoroughly 

3. feel free to write or draw on the blank paper below the problem 

4. you may ask for general assistance, but the interviewer cannot 
suggest solutions, etc. to the problem 

If you are confused in any way as to what these directions instruct 
you to do, please ask at this time. This procedure will be repeated for 
each of the three problems. 
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APPENDIX D 

EXAMPLES OF ECONOMIC CONCEPTS USED IN RESPONSES 
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Economic Concepts Used by Respondents 



supply and den\and 


fixed exchanffe rates 


equilibriun\ price 


flexible exchange rates 


market clearing price 


currency 


unemployment 


real value of the dollar 


labor supply 


short-run 


labor supply curve 


long-run 


quantity of labor adjustment 


distribution of income 


wage bill 


Gini co-efficient 

A A ^ Jl V* AAA V*A A ^ ^ 


efficiency 


tariffs 


net value 


quotas 


assets 


business 


economic price 


cycle 


price 


elasticity of demand 


markets 


speculation 


deflation 


nominal interest rates 


inflation 


real interest rates 


money supply 


contractionary policy 


budget deficit 


expansionary policy 


the Federal Reserve system 


trade balance 


full employment 


industrial policy 

JT J 


fiscal policy 


opportunity costs 


competition 


productivity 


producers 


marginal productivity 


consumers 


capital 


price levels 


free trade 


subsidies 




exports 




dumping 




externalities 




anti-trust 




trade barriers 




scale economies 




interventionist 




comparative advantage 




human capital 




real wage rates 




minimum wage 




wage floor 




investment expenditures 




aggregate demand 




aggregate supply 




fiscal policy 




monetary policy 




taxation 
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APPENDIX E 

LIST OF ECONOMIC MODELS USED BY RESPONDENTS 
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E xamples of Economic Models Used by Respondents 



labor market model 

capital market model 

exchange rate model 

Keynesian AD model 

Industrial organization/ policy model 

public choice model 

monetarist model 

general market model 

IS-LM model 

micro price theory model 

long-run growth model 



